In general, the present invention relates to a three-dimensional-picture-generating apparatus, a three-dimensional-picture-generating method and a program-presenting medium with regard to texture picture pasting techniques for three-dimentionally-shaped model. More particularly, the present invention relates to a three-dimensional-picture-generating apparatus and a three-dimensional-picture-generating method that are capable of generating a real three-dimensional picture with little unnaturalness by improving processing to paste texture pictures each taken at a virtual point of sight.
There are known measurement techniques applied to an object having a three-dimensional shape which is treated as an object of measurement. The measurement techniques include a method whereby a beam is radiated by a beam source to the object of measurement and a time it takes till a beam reflected by the object reaches the source is measured. In another measurement technique, a pattern beam having a slit shape is radiated to an object of measurement and the shape of the pattern beam projected on the object of measurement is inspected. In accordance with a further measurement technique known as a stereo picture method for measuring a distance, at least 2 cameras are used and a corresponding point between their pictures is identified to find a parallax in a measurement of a distance.
A picture reflecting actual colors of a measurement object is pasted on a three-dimensional model representing the measurement object""s distance data obtained as a result of measurements using a variety of such methods to generate a real three-dimensional picture. An actual picture of the measurement object is called a texture picture. A process of pasting a texture picture on a three-dimensional shape is referred to as texture mapping.
In general, a picture pasted on a surface of a three-dimensional model is a two-dimensional bit-map picture or an image-file picture. A texture picture comprising a two-dimensional bit-map picture and an image-file picture is pasted and combined on a surface of a three-dimensionally-shaped model created by using a three-dimensional graphic technique or a surface of a substance""s three-dimensional shape obtained by using a three-dimensional-shape measurement apparatus to display a three-dimensional substance. By pasting the picture of bricks on such a surface, for example, a natural three-dimensional wall can be expressed. By pasting a picture of a world map on a spherical body, a cubic earth can be expressed.
FIG. 1 is an explanatory diagram showing a general texture-mapping technique. A three-dimensionally-shaped model 101 shown in the figure is a model based on distance data obtained as a result of measurements using typically the stereo picture method described above or the like. A two-dimensional picture (a texture) 102 is a real picture reflecting visual-sense color data or the like obtained as a result of photographing from a certain point of vision. Texture mapping is a process of pasting the two-dimensional picture (the texture) 102 on the three-dimensionally-shaped model 101. By carrying out the texture-mapping process, a real three-dimensional expression is possible. It should be noted that a wire-frame display shown in the figure shows a planar area on which the texture obtained from the three-dimensionally-shaped model 101 is pasted.
So far, there have been studied a variety of technologies for pasting a texture picture on a substance""s three-dimensional shape obtained by adoption of the stereo picture method or by using a three-dimensional measurement apparatus utilizing both a laser beam and a projection beam or the like. Assume that texture pictures seen from a plurality of visual points are pasted on corresponding areas of a measurement object. In this case, if the texture pictures are taken under different beam-source conditions, there will be differences in color tone among the texture pictures. The differences in color tone are caused by differences in beam-source condition which prevail when the texture pictures are taken individually. The differences in color tone result in an unnatural combination of colors in a picture if the resulting picture is produced by merely pasting the texture pictures. In order to solve this problem, there is provided a texture-mapping technique to produce a higher picture quality whereby beam-source conditions under which the texture pictures are taken are estimated, and the texture pictures are compensated for the differences in color tone on the basis of a result of the estimation. An example of such a technique is described in a reference authored by Imari Satoh, Youichi Satoh and Katsumi Ikeuchi with a title of xe2x80x9cAll-Azimuth Measurements of Beam-Source Environments and Superposition of Real Pictures on a Virtual Substance Based on Results of the Measurements,xe2x80x9d a thesis of the Academic Society of Electronic, Information and Communication Engineers D-II, Vol. J81-D-II, No. 5, pp. 861-871, 1998.
When a picture of outdoor scenery or a building is taken by using an apparatus such as a digital still camera or a digital video camera in general, however, the condition of a source radiating a beam is not easy to estimate. It is thus difficult to implement texture mapping to produce a high picture quality by using a plurality of taken pictures, that is, a plurality of texture pictures. In order to avoid unnaturalness caused by use of a plurality of texture pictures, for example, there is also provided a texture-mapping technique using only one texture picture as shown in FIG. 2. FIG. 2 is a diagram showing a configuration for implementing a texture-mapping process by pasting a particular picture of a proper area of a two-dimensional picture (texture picture) 202 on a wire frame of a three-dimensionally-shaped model 201.
With a technique of using only 1 particular picture as described above, however, there is raised a problem of a deteriorating texture picture quality caused by a change in point of vision. Assume for example that the acquired two-dimensional picture (texture picture) 202 is based on the picture of the front surface 203 of a cup shown in FIG. 2 and an end 204 thereof. In this case, there will be resulted in a difference in picture quality between the front surface 303 and the end 204 of the cup. This difference in picture quality causes deterioration of the picture quality.
As a technique of solving the deterioration of the picture quality described above, there is provided a method of selecting a texture picture taken from a photographing visual point closest to the virtual point of view and pasting the selected texture picture on a three-dimensionally-shaped model. This method is described by referring to FIG. 3. In the method shown in FIG. 3, a texture picture of a measurement object 301 is taken from a plurality of visual points, namely, photographing visual points 1 to n. If a virtual point of vision is set, a photographing visual point closest to the virtual visual point is selected, and a texture picture taken from the selected photographing visual point is pasted on a three-dimensionally-shaped model of the measurement object 301. In the example shown in FIG. 3, the RGB values of a virtual visual point are used as those of the photographing visual point 1 or 2. Notation xcex81 denotes an angle formed by directions from the virtual visual point and the photographing visual point 1, whereas notation xcex82 denotes an angle formed by directions from the virtual visual point and the photographing visual point 2. If xcex81 less than xcex82, the RGB values of the photographing visual point 1 are selected. If xcex81 greater than xcex82, on the other hand, the RGB values of the photographing visual point 2 are selected. By adopting this method, the picture quality is improved. With this method, however, it is difficult to maintain concentration contiguity between pictures and, if the point of vision changes, unnaturalness of the picture results.
For a plurality of texture pictures usable for a surface, there has been a technique known as a VDTM (View-Dependent Texture Mapping) for expressing a real three-dimensional substance by carrying out a selection and synthesis process on a texture picture accompanying a movement of a visual point. This technique is described in a reference authored by Paul E. Debevec, Camillo J. Taylor and Jitendra Malik with a title of xe2x80x9cModeling and Rendering Architecture from Photographs: A hybrid geometry-and image-based approach,xe2x80x9d ACM SHIGGRAPH ""96 Proceedings, pp. 11-20, 1996. This VDTM technique is different from the conventional CG technique of carrying out a texture-mapping process using a texture picture in that, with this VDTM technique, a change in visibility caused by a movement of a visual point for a portion irreproducible as a three-dimensional shape can be reproduced. With this VDTM technique, however, there is raised a problem of an increasing amount of picture data stored for improving the picture quality. In addition, the VDTM technique also imposes a limitation requiring that the conditions of the beam source for the texture pictures be unchanged.
In order to reduce the amount of picture data stored for improving the picture quality, there has been developed a mapping technique whereby only picture data of a visible portion is transmitted each time the point of vision changes. This technique is described in a reference authored by S. Horbelt, F. Jordan and T. Ebrahimi with a title of xe2x80x9cView-Dependent Texture Coding for Transmission of Virtual Environment,xe2x80x9d Proceedings of the 1998 IEEE International Symposium on Circuits and Systems, Vol. 5, No. 6, pp. 498-501, 1998. However, this technique has a shortcoming that data is accessed with a low degree of efficiency due to the fact that data is transmitted each time the point of vision changes on the user side.
In addition, there are also provided techniques of interpolation among pictures. These techniques take advantage of association of a picture with another, which are taken at reconfiguration of a three-dimensional shape. These techniques are each proposed for reducing the amount of picture data. They are described in a reference authored by Shenchang Eric Chen and Lance Williams with a title of xe2x80x9cView Interpolation for Image Synthesis,xe2x80x9d ACM SHIGGRAPH ""93 Proceedings, pp. 279-288, 1993 and a reference authored by S. Steven, M. Seitz and Charles R. Dyer with a title of xe2x80x9cView Morphing,xe2x80x9d ACM SHIGGRAPH ""96 Proceedings, pp. 21-30, 1996. With these methods, however, there is raised a problem of picture-quality deterioration and unnaturalness depending on the way a picture is taken at a photographing time.
In the mean time, there has been studied a method of correcting a difference in color tone among pictures by adoption of a technique of estimating conditions of a beam source that prevail at the time a texture picture is taken. The technique is described in a reference authored by Imari Satoh, Youichi Satoh and Katsumi Ikeuchi with a title of xe2x80x9cEstimation of Real Radiation Conditions Based on Soft Shadows of a Real Substance,xe2x80x9d a research report of the Society of Information Processing Engineers, 98-CVIM-110, 1998. However, this method imposes a limitation that a substance determined on a picture must be moved in order to carry out correction processing. With this method, it is impossible to estimate conditions of a beam source for a picture taken in the open air.
In view of the foregoing, the present invention has been made, and an object of the present invention is to provide a three-dimensional-picture-generating apparatus and a three-dimensional-picture-generating method capable of providing a high-quality picture-mapping technique for improving the picture quality of a texture while decreasing an amount of picture data to be stored. To put it concretely, it is an object of the present invention to provide a three-dimensional-picture-generating apparatus and a three-dimensional-picture-generating method, which are capable of producing a three-dimensional expression with a high picture quality by applying a smoothing method and picture concentration correction of an image base without creating a model of a beam source; creating a texture picture using a picture with a highest resolution on each patch surface; and subjecting the picture created at the highest resolution to the smoothing method and the picture concentration correction to make differences in color tone between pictures less striking wherein the differences in color tone between pictures are caused by differences in beam-source conditions.
According to a first aspect of the present invention, there is provided a three-dimensional-picture-generating apparatus for generating a three-dimensional picture by pasting texture pictures on a three-dimensionally-shaped model, which provides for the selection of a picture with a highest resolution from among a number of pictures obtained as a result of photographing an object of measurement from different camera points of view for each of patch surfaces composing a three-dimensional picture; and a texture mapping function including applying the selected pictures to the respective patch surfaces as texture pictures.
In the three-dimensional-picture-generating apparatus, preferably the selection of a picture with a highest resolution among a number of pictures for any particular one of the patch surfaces involves a comparison of a normal vector of the particular patch surface with each of a number of photographing visual-point direction vectors; and may select a picture taken from a photographing visual-point direction forming a smallest angle with the normal vector of the particular patch surface.
Preferably, the three-dimensional-picture-generating apparatus may include a processor for carrying out processing to correct concentrations of a number of pictures taken from different camera photographing points of view, wherein the processing to correct concentrations is configured to compute an average and a variance of concentrations for each patch-surface unit or each set area unit of the pictures taken from different camera photographing points of view; and perform normalization.
Preferably, the three-dimensional-picture-generating apparatus may include a processor for carrying out processing to correct a texture-picture shift between patch surfaces, wherein the processing to correct a texture-picture shift is configured to compute a difference in concentration between adjacent patch surfaces on a border between the patch surfaces; detect the texture-picture shift; further divide each of the patch surfaces on the basis of the detected texture-picture shift; and perform re-mapping.
Preferably, the three-dimensional-picture-generating apparatus may include a processor for carrying out processing to correct a texture-picture shift between patch surfaces, wherein the processing to correct a texture-picture shift is configured to compute distributions of concentrations of adjacent patch surfaces on a border between the patch surfaces; and change coordinates of end points of either of the patch surfaces to coordinates that provide a minimum total sum of differences which is computed with respect to pixels on the distributions.
Preferably, the three-dimensional-picture-generating apparatus may include a processor for carrying out processing to correct a texture-picture shift between patch surfaces, wherein the processing to correct a texture-picture shift is configured to compute an error in concentration between each two adjacent patch surfaces on a border between the patch surfaces for a number of patch surfaces composing a three-dimensional picture; perform processing to correct a shift under a condition that an error is greater than a threshold value determined in advance for a largest value among a number of computed errors in concentrations; and complete said processing to correct a shift under a condition that errors in concentration which are computed between each two adjacent patch surfaces on a border between the patch surfaces for a number of patch surfaces composing a three-dimensional picture all do not exceed the threshold value.
Preferably, the three-dimensional-picture-generating apparatus may include a processor for carrying out processing to smooth concentrations on a border between patch surfaces, wherein the processing to smooth concentrations is configured to compare a normal vector of the particular patch surface with each of a number of photographing visual-point direction vectors; select a picture taken from a photographing visual-point direction forming a smallest angle with the normal vector of the particular patch surface; paste the selected picture on the particular patch surface; create as many texture development diagrams each comprising a texture picture pasted on a number of patch surfaces by superposition as a number of texture pictures pasted on patch surfaces; superpose the texture development diagrams on each other; and smooth concentrations of each of the patch surfaces.
According to second aspect of the present invention, there is provided a three-dimensional-picture-generating method for generating a three-dimensional picture by pasting texture pictures on a three-dimensionally-shaped model, which includes the steps of: selecting a picture with a highest resolution among a number of pictures obtained as a result of photographing an object of measurement from different camera points of view for each of patch surfaces composing a three-dimensional picture; and carrying out texture mapping by applying the selected pictures to the respective patch surfaces as texture pictures.
In the three-dimensional-picture-generating method, preferably, the step of selecting a picture with a highest resolution for any particular one of the patch surfaces may include the steps of: comparing a normal vector of the particular patch surface with each of a plurality of photographing visual-point direction vectors; and selecting a picture taken from a photographing visual-point direction forming a smallest angle with the normal vector of the particular patch surface.
Preferably, the three-dimensional-picture-generating method may include the step of carrying out processing to correct concentrations of a number of pictures taken from different camera photographing points of view, wherein the processing to correct concentrations is carried out by computing an average and a variance of concentrations for each patch-surface unit or each set area unit of the pictures taken from different camera photographing points of view and performing normalization.
Preferably, the three-dimensional-picture-generating method may include the step of carrying out processing to correct a texture-picture shift between patch surfaces, wherein the processing to correct a texture-picture shift is carried out by computing a difference in concentration between adjacent patch surfaces on a border between the patch surfaces; detecting the texture-picture shift; further dividing each of the patch surfaces on the basis of the detected texture-picture shift; and performing re-mapping.
Preferably, the three-dimensional-picture-generating method may include the step of carrying out processing to correct a texture-picture shift between patch surfaces, wherein the processing to correct a texture-picture shift is carried out by computing distributions of concentrations of adjacent patch surfaces on a border between the patch surfaces; and changing coordinates of end points of either of the patch surfaces to coordinates that provide a minimum total sum of differences which is computed with respect to pixels on said distributions.
Preferably, the three-dimensional-picture-generating method includes the step of carrying out processing to correct a texture-picture shift between patch surfaces, wherein the processing to correct a texture-picture shift is carried out by computing an error in concentration between each two adjacent patch surfaces on a border between the patch surfaces for a number of patch surfaces composing a three-dimensional picture; performing processing to correct a shift under a condition that an error is greater than a threshold value determined in advance for a largest value among a number of computed errors in concentrations; and completing said processing to correct a shift under a condition that errors in concentration which are computed between each two adjacent patch surfaces on a border between the patch surfaces for a number of patch surfaces composing a three-dimensional picture all do not exceed said threshold value.
Preferably, the three-dimensional-picture-generating method may include the step of carrying out processing to smooth concentrations on a border between patch surfaces, wherein the processing to smooth concentrations is carried out by comparing a normal vector of the particular patch surface with each of a number of photographing visual-point direction vectors; selecting a picture taken from a photographing visual-point direction forming a smallest angle with the normal vector of the particular patch surface; pasting the selected picture on the particular patch surface; creating as many texture development diagrams each comprising a texture picture pasted on a plurality of patch surfaces by superposition as a plurality of texture pictures pasted on patch surfaces; superposing the texture development diagrams on each other; and smoothing concentrations of each of the patch surfaces.
As described above, in the three-dimensional-picture-generating apparatus and three-dimensional-picture-generating method in accordance with the present invention, a texture picture is created by using a picture with a highest resolution for each patch surface, and a concentration correction method as well as a smoothing method are applied to processing to correct differences in concentration, which are caused by differences in beam-source condition, between pictures or between patch surfaces. As a result, a texture mapping process producing a high picture quality can be carried out.
In addition, in the three-dimensional-picture-generating apparatus and three-dimensional-picture-generating method in accordance with the present invention, differences in concentration, that is, differences in RGB values, on each patch surface are corrected by carrying out correction processing based on concentration normalization and reference picture concentration interpolation for each patch surface to generate a texture picture. In addition, by adoption of a smoothing method based on picture superposition using development diagrams of patch surfaces, a joint or a boundary area between patch surfaces each pasted with a created texture picture can be made smooth.
According to a third aspect of the present invention, there is provided a program-presenting medium for presenting a formatted computer program to be executed by a computer system for generating a three-dimensional picture by pasting texture pictures on a three-dimensionally-shaped model, which computer program includes the steps of: selecting a picture with a highest resolution among a number of pictures obtained as a result of photographing an object of measurement from different camera points of view for each of patch surfaces composing a three-dimensional picture; and carrying out texture mapping by applying the selected pictures to the respective patch surfaces as texture pictures.
The program-presenting medium described above may be a medium which presents a computer program in a computer-readable format to a typical general-purpose computer system capable of executing a variety of program codes. The medium may be a recording medium such as a CD (Compact Disc), an FD (Floppy Disc) and an MO (Magneto-Optical) disc as well as transmission media such as a network. The type of the program-presenting medium is not specially restricted.
Such a program-presenting medium for presenting a computer program defines a structural or functional cooperative relation between a computer system and the program-presenting medium. The relation is used for implementing predetermined functions of the computer program in the computer system. In other words, by installing the computer program into the computer system through the program-presenting medium, the computer system is capable of displaying a cooperative effect, allowing the same effects as other aspects of the present invention to be obtained.
The above and other objects, features and advantages of the present invention as well as the manner of realizing them will become more apparent whereas the invention itself will be best understood from a careful study of the following detailed description and appended claims with reference to attached drawings showing the preferred embodiments of the invention.